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Tutorial  Objective 


This  tutorial  will  present  Return-on-lnvestment  (ROI)  concepts  and 
processes  that  can  be  used  to  determine  the  ROI  of  a  model  or  simulation 
in  any  functional  area  by  enabling  the  student  to: 

a  apply  ROI  terms,  guidelines  and  formulas  to  their  own  M&S  projects 

»  evaluate  all  costs  /  resources  associated  with  an  M&S  program 

a  identify  alternative  methods  of  achieving  goals 


a  calculate  the  ROI  of  an  M&S  program  based  on  quantitative  and 
qualitative  criteria 


Tutorial  Outline 


1.  Introduction  to  Return-on-lnvestment 

1.1.  What  Is  ROI  and  Why  Is  It  Important? 

1.2.  ROI  for  Modeling  and  Simulation 

2.  Process  for  Quantifying  M&S 

2.1.  Requirements  Definition 

2.2.  Nine  ROI  Factors  for  M&S 

3.  Evaluating  an  M&S  Application 

3.1.  Designing  Metrics  for  the  Nine  Factors 

3.2  Functional  Implications  of  the  Nine  Factors 

3.2.1.  Quantitative  Metrics 

3.2.2  Qualitative  Metrics 


M&S  -  Negative  or  Positive  ROi? 


Simulation  and  acquisition  programs  (that  used  M&S)  failures  have  raised 
concerns  about  why  these  investments  so  often  fail  to  live  up  to 

expectations. 


M&S  in  the  Gulf  War 
Casualties  Count 


Rise  and  Fall  of 
JSIMS 


What  is  RO! ? 


A  performance  measure  used  to  evaluate  the  efficiency  of  an  investment  or  to 
compare  the  efficiency  of  a  number  of  different  investments. 


Investment 


Alternative(s) 


ROI 


People 


Money 


Individual 

On 

Individual 


Small 


Multiple 


Individual 


Organization  Organization 


Virtual 

Simulation 


mm 


j  '■  v 


Constructive 

Simulation 

Live 

Simulation 


Time 


http://dictionary.reference.com/browse/ROI 


Why  Should  /  Care  about  ROI? 


Justifies  the  investment  to  decision  makers 


Provides  justification  for  oversight  of  funding 
the  investment  throughout  its  life  cycle 


Establishes  a  baseline  to  monitor, 
measure  and  evaluate  the  investment 


Questions  for  an  ROi  Analysis 


Effectiveness:  How  much  — bancfor  the  buck”  will  we  get  out  of  this  project? 


Impact:  Will  the  benefits  to  USAF,  DoD,  federal  government,  country  or  society 

justify  the  overall  investment  in  this  project? 


Questions  for  an  ROi  Analysis 


Efficiency:  Is  this  the  most  we  can  get  for  this  much  investment? 


Alternatives  to  Bluetooth 


Bluetooth 

HomeRF 

IEEE  802.11b 
(Wi-Fi) 

IrDA 

Primary  Use 

Cable 

replacement 
and  ad  hoe 
devica-to- 
d  evict 
connections 

Home  or 

small  office 
LANs 

Corporate  or 
campus  LANs 

Cable 

replacement  and 
ad  hoc  device-to- 
dsvice 
connections 
(narrow  angle) 

Max  Speed 

1  Mbps 

10  Mbps 

11  Mbps 

4  Mbps 

Range 

30  feet 

150  feet 

300  feet 

3  feet 

Through  walls 

Yes 

Yes 

Yes 

No 

Requires  base 
station 

No 

No 

Yes 

No 

Interference 

susceptibility 

Medium 

Medium 

High 

None 

\Power  Reqs 

Low 

High 

High 

Low 

\^to  build 

$8  now 

$70  $120 

$100-$300 

$2  y 

Financial:  Can  we  afford  this?  Will  it  pay  for  itself? 


Conducting  an  ROi  Analysis 


introduction  to 

Simulation  and 
Risk  Analysis 


Payment  Methods  i  Merchant  Account  Options  |  Checkout  Descriptions  |  Custom  Field(s)  Other  Options  1 


Select  one  of  available  payment  method  (s): 

Select  one  of  the  activated  payment  method(s): 

r  Open  Account 

["  Credit  Card  (Modify) 

r  Institutional  Purchase  Order 

f  Cash  (Modify) 

r  Personal  Check 

T  Online  Check  (Modify) 

r  Gift  Certificate 

T  Custom  Payment  Method  (Modify) 

r  Money  Orders 

F  International  Bank  Draft  (Modify) 

r  UPS-Cash  On  Delivery 

f  PayPal  (Setup) 

Activate  ==> 

<==  Deactivate 

ROI  a  rather  diverse  collection  of 
methods,  skills,  tools,  activities 
and  ideas. 


No  single  — rigU”  way  to 
conduct  a  ROI  analysis. 


■  ROI  is  not  a  silver  bullet  -  nor  is  there  a  Consumer 

Report  for  ROI  products  and  services. 


Design  Metrics 


Remember  the  perspective  of  who  the  ROI  is  for. 


Internal  to  the 
Program 


Program  Manager 
Using 

Organizations 


User 


Tragedy  at  Fort  Hood 


External  to  the 
Organization 


Press 

Other 


Environmentalist 


Areas  for  M&S  Return  on  Investment 


Efficiency  Risk  Reduction  Effectiveness 


Money  Environment  Lives  Time 


Areas  for  M&S  Return  on  Investment 


Reuse 


Extended  Air 


Defense  Simulation  (EADSIM) 


First  Deployed  in  1989 


Used  by: 

•  Combat  developers 

•  Materiel  developers 

•  Operational  commanders 


Interfaces: 

•  Aggregate  Level  Simulation  Protocol  (ALSP) 

•  Distributed  Interactive  Simulation  (DIS) 

•  High  Level  Architecture  (HLA) 


EADSIM  used  during  DESERT  SHIELD/DESERT  STORM: 

•  To  analyze  attrition,  Suppression  of  Enemy  Air  Defense  (SEAD)  missions  and 

refueling  operations 

•  BGen  Glosson  stated  that  EADSIM  "saved  lives  and  equipment." 

•  32nd  Army  Air  Defense  Command  used  EADSIM  to  analyze  proper  positioning  of 

PATRIOT  in  Israel  and  Turkey 

EADSIM  is  now  being  used  at  more  than  390  subscriber  sites  around  the  world. 

http://www.smdc.army.mil/FactSheets/EADSIM.pdf 


Areas  for  M&S  Return  on  Investment 


B2  Bomber 

•  Limited  Number  of  Aircraft -21  Operational  (1  Test) 

•  Operation  Construct  -  Stateside  Basing 

•  Construct  Leads  to  Long  Missions 

•  Longest  Single  Sortie  was  44  hours 

•  Crew  -  2  Pilots 

•  Cost  to  Fly  Aircraft 

•  Train  via  Simulation 

"B2  pilots’  greatest  challenge  is  endurance...  in  training  they  spend  as 
long  as  50  hours  in  simulators.”  Maj  Gen  Przybyslawski 


Readiness 


Areas  for  M&S  Return  on  Investment 


Efficiency 


ROVER 

Remote  Operations 
Vehicle  Enhanced 
Receiver 


Provides  Joint  Terminal  Air  Controller  (JTAC)  with  Predator  video  to  provide 
terminal  attack  control  without  direct  eyes  on  target 


Areas  for  M&S  Return  on  Investment 


Risk  Reduction 


Jet  Blue 

— flat  is  a  very  difficult  maneuver, 
especially  since  pilots  are  not 
given  simulator  time  to  practice 
it.  As  of  Monday  this  event  will  be 
part  of  simulator  training.” 


http://www.airlinesafety.com/editorials/JetBlueLAX.htm 


Areas  for  M&S  Return  on  Investment 


Joint  Strike  Fighter  Applications 
Arnold  Engineering 
and  Development  Center 


Effectiveness 

Wind  Tunnel  Effects 
Airframe  Loads 
Carriage  Loads 
Store  Separation 

-  Internal  Carriage 

-  External  Carriage 
Fuel  Tank 

-  Design 

-  Loads 

-  Jettison 

Aircraft  lift  fan/secondary  inlet  design 


Eliminated  tests  for  high  speed  data 
Improved  data  quality  and  reduced  risk 
Computed  trajectories  beyond  tunnel  hardware 
movement  constraints 

Screened  test  configurations,  reducing  testing 
costs 

Total  savings  =  $  Millions 


http://www.afmc.af.mil/news/story.asp?id=1 2301 8039 


Areas  for  M&S  Return  on  Investment 


Lives 


•  Oxygen  tank  No.  2  blew  up 

•  Command  modules  normal  supply  of  electricity,  light  &  water  was  lost 

•  200,000  miles  from  Earth 


•  New  procedures  had  to  be  written  and  tested 

•  A  coast-to-coast  network  of  simulators,  computers  and  experts  was  formed 

•  Everything  was  tested  in  the  simulator  before  being  passed  up  to  the  crew 


http://liftoff.msfc.nasa.gov/Academy/History/APOLLO-13/MISSION-REPORT.HTML 


Areas  for  M&S  Return  on  Investment 


Electric  Boat,  by  using  M&S,  took  7  years  off  development  of  the  submarine. 

http://www.defense-update.com/newscast/01 07/news/01 01 07_virginia.htm 


Areas  for  M&S  Return  on  Investment 


•  Invested  $1.4  billion  to  purchase  additional  simulators  and  upgrade  existing  ones. 


•  Estimates  aircraft  flight  hours  will  be  reduced  by  270,000  hours  over  the  next  6.5  years. 

•  Save  $2.3  billion  in  aircraft  fuel,  airframe  use,  wear  and  tear  and  aircraft  maintenance. 

http://www.amc. af.mil/news/story.asp?id=1 23031 427 


Areas  for  M&S  Return  on  Investment 


Endangered 

Species 


Ft  Hood  Environmental  Restrictions 
199,541  total  land 

154,053  acres  with  training  restrictions  -  77%  of  training  land 
30,827  acres  of  maneuver  land  without  training  restrictions 

GAO-Q2-727T  Military  Training 
Encroachment  had  affected  every  training 
range’s  capabilities,  requiring  work¬ 
arounds — or  adjustments  to  training 
events.  A  more  complete  assessment  of 
training  resources  should  include 
assessing  the  potential  for  using  virtual  or 
constructive  simulation  technology  to 
augment  live  training. 


Clean  Water  - 
Erosion  Control 


Environment 


Due  to  restrictions,  weapons  or  testing  threats  are  prohibited. 
Defense  Threat  Reduction  Information  Analysis  Center 


http://www.gao.gov/htext/d03621t.html 


Areas  for  M&S  Return  on  Investment 


Reuse 


Efficiency 


Readiness 


Lives 


V'_. 

Risk 

Reduction 


Effectiveness 


Time 


Money 


'V 

«§j 

L 

Environment 


Training 


Increase  Hof 
users  and 
applications 
fbrthe 
simulation 

Proficiency 
level  after  a 
simulation 
event 

Increased 
preparedness 
to  a  greater 
numberof 
situations 

Reduced 
deaths  & 
injuries 
through 
simulations 

Reduced 
accidents 
during  high- 
risk  training 

Increased 
performance 
in  real-world 
situation 

Optimize 
training  for 
the  audience 

Optimize  or 
reduce  the 
cost  of 
training 

Use 

simulation 
for  training 
that  can  not 
be  done 

Acquisition 

Increase  Hof 
users  and 
applications 
forthe 
simulation 

Lower  cost 
due  to 
simulation 

use 

Faster 
delivery  of 
new  materials 
to  the 
warfighter 

Simulation 
testing  prior 
to  human 
conducting 
live  testing 

Remove 
more  of  the 
unknowns  in 
the  testing 
cycle 

Able  to,  as  a 
minimum , 
keep  project 
on  time  and 
budget 

Maintain  or 
reduce 
development 
time 

Maintain  or 
reduce 
acquisition 
costs 

Offset 
restrictions 
caused  by 
environment 
constraints 

Analysis 

Increase  Hof 
users  and 
applications 
fbrthe 
simulation 

Greater 
accuracy 
based  on  the 
use  of  a 
simulation 

Increase  the 
timeliness  of 
analytical 
information 

Removing 
some  areas 

of  human 
error  prior  to 
decisions 

Look  at 
larger 
numberof 
whatifs 

Fill  in  void 
when  subject 
matter 
experts  are 
unavailable 

Provide 

answers 

faster 

Reduce  costs 
for 

development 
of  analytical 
products 

Analyze 
situations 
that  can  not 
be  done  live 

Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Set  Standards 
of 

Performance 


Design 

M&S 

Application 


Collect 

Data 


Adjust  Throughout  the  Process! 


If  Needed,  Initiate 
Corrective  Action 

1 

Determine  ROI 

Analyze  Data 

Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Adjust  Throughout  the  Process! 


Determine  Requirements 
Describe  the  Problem 


Within  the  United  States  civilian  and  military  aviation  system  there  are  numerous  crisis  situations  that 

occur  each  year  that  receive  both  local  and  national  attention. 


IT  JUST 
EXPLODED 
AND  LIT  UP 
THE  SKY' 

LJ 

t 

» 

'  II 

7  r  ^  *v 

49  Dead  as  Commuter  Plane  Crashes  Into  Buffalo  Suburb 

Determine  Requirements 
Example  -  Perspective  Implications 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Acquisition  Example  -  How  do  I  design  a  safer  aircraft? 


Analysis  Example  -  How  do  I  react  in  the  event  of  a  crisis  at  an  airport? 


ORLANDO  INTERNATIONAL  AIRPORT  (MCO) 


Determine  Requirements 
Example  -  identify  Users 


Training  Users 


Acquisition  Users 


Analysis  Users 


Determine  Requirements 
Example  -  Describe  the  Problem 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Pilot 


Flight 

Attendant 


Ground 

Crew 


TSA 


Ground  Air  Traffic 

Controller  Controller 


Local  Medical  On  Site 
Responders  Responders 


Ticket 

Agents 


Local  SWAT 
Teams 


Local  Police 
Aviation 


Medevac 

Teams 


Aircraft  crew  members  need  to  recognize,  evaluate  and  respond  to  emergency  situations 
on  an  aircraft  with  an  appropriate  crew  resource  management  course  of  action. 


Process  for  Quantifying  M&S 


Determine 

Requirements 


Set  Standards 
of 

Performance 


Document  the  Process  and  Results! 


Adjust  Throughout  the  Process! 


Standards  of  Performance 


The  manner  in  which  something  or  somebody  functions,  operates 
or  behaves  in  terms  of  established  criteria 


Standards  of  Performance 

Example 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


IX.  AREA  OF  OPERATION:  EMERGENCY  OPERATIONS 

A.  TASK:  EMERGENCY  APPROACH  AND  LANDING  -  (ASEL  and  ASES)  REFERENCES: 
FAA-H-8083-3;  POH/AFM. 

Objective.  To  determine  that  the  applicant: 

1 .  Exhibits  knowledge  of  the  elements  related  to  emergency  approach  and  landing 
procedures. 

2.  Analyzes  the  situation  and  selects  an  appropriate  course  of  action. 

3.  Establishes  and  maintains  the  recommended  best  glide  airspeed,  ±10  knots. 

4.  Selects  a  suitable  landing  area. 

5.  Plans  and  follows  a  flight  pattern  to  the  selected  landing  area  considering  altitude,  wind, 
terrain,  and  obstructions. 

6.  Prepares  for  landing,  or  go-around,  as  specified  by  the  examiner. 

7.  Follows  the  appropriate  checklist. 


Standards  of  Performance 

Example 


Acquisition  Example  -  How  do  I  design  a  safer  aircraft? 


The  FAA  provides  the  following  categorizations  of  aircraft. 

1. Categorization  by  Stall  Speed  (which  determines  the  basis  of  landing  or  approach  speed):  Table  1  -  Aircraft  Approach  Category  Source:  FAA, 
1976,  United  States  standards  for  Terminal  Instrument  Procedures,  3d  ed,  FAA  Handbook  8260. 3B) 

Aircraft  Category 

1.3  Times  the  Stall  Speed  in 
Knots 

Maximum  Speed  (Circling  Approaches) 

Typical  Aircraft  in  This  Category 

A 

less  than  91  knots 

90  knots 

small  single  engine 

B 

91  to  120  knots 

120  knots 

small  multi  engine 

C 

121  to  140  knots 

140  knots 

airline  jet 

D 

141  to  165  knots 

164  knots 

large  jet/military  jet 

E 

above  166  knots 

special  military 

Standards  of  Performance 

Example 


Analysis  Example  -  How  do  I  react  in  the  event  of  a  crisis  at  an  airport? 


•  Current  FAA  standards  for  rescue  and  firefighting  equipment  are  included  in  both  FARs  and 
Advisory  Circulars. 

•Aircraft  rescue  and  firefighting  indexes  are,  in  accordance  with  FAR  Part  139.315, 
designated  by  an  index  letter  (i.e.,  A,  B,  C,  D,  or  E)  that  represents  the  size  of  the  largest 
aircraft  the  airport  is  prepared  to  handle  in  the  event  of  a  fire  or  rescue  situation. 

•  Part  139.315  further  states  that  — ilfhere  are  five  or  more  average  daily  departures  of  air 
carrier  aircraft  in  a  single  index  group,  the  longest  index  group  with  an  average  of  5  or  more 
daily  departures  is  the  index  required  for  the  airport.  In  addition,  if  there  are  less  than  five 
average  daily  departures  of  air  carrier  aircraft  in  a  single  index  group  serving  that  airport,  the 
next  lower  index  from  the  longest  index  group  with  an  air  carrier  aircraft  in  it  is  the  index 
required  for  the  airport.” 


Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Adjust  Throughout  the  Process! 


Conducted  prior  to  the  beginning  of  the  event. 
Point  of  comparison  for  monitoring  and  evaluation  data. 


Predictive 

Evaluation 

Ensuring  Training  Delivers 
Business  and  Organizational  Results 


Dave  Basarab 


Focuses  on  the  intended  outcomes  of  a 

project. 


Figure  1 :  Pathways  to  Improved  Maternal  Health  Outcomes 
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Accounts  for  secondary  outcomes  and  assumptions. 


Source:  PR, SR  Sourcebook,  Claetnn,  et  al. 


Baseline  Example 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Collect  information  on  the  current  state  of  formal  and  informal  training  or  education  for 
employees  who  would  deal  with  an  aviation  incident  /accident. 


Conduct  interviews  to  gather  information  to  benchmark  the  levels  of 
knowledge,  experience  and  training  for  each  individual. 


Conduct  a  qualitative  survey  on  individual  perception  of  being 
prepared  for  an  aviation  incident  /accident. 


Baseline  Example  -  Survey 


B  National  Aviation  Operations  Monitoring  Service  (NAOMS) 

NASA  Airline  Study 


•  Air  Carrier  Questionnaire:  Section  B  -Safety  Events” 


•Aircraft  mechanical  &  equipment  (ER1-ER7)  maintenance,  equipment  failures,  fire,  smoke 


•  Turbulence:  Wake  and  en  route  (TU1-TU2): 

-  Weather  (WE1-WE6)  Icing,  diversions,  ATC  Wx  issues  &  windshear 

-  Passenger-Related  Events  (CP1-CP3)  Disruption,  medical  emergency 
-Airborne  Conflicts  (AC1-AC3)  Bird  strikes,  NMACs 

-  Ground  Events  (GE1-GE10)  Near  collisions,  hydroplaning,  off  runway  events 


•  Aircraft  handling  (AH1-AH15)  Variety  of  pilot-related  issues: 

-  Overweight,  stalls,  unusual  attitudes,  tail  strikes,  etc. 

-Altitude  Deviations  (AD1-AD2)  Descend  below  MSA,  deviation  from  assigned  altitude 
-Air  Traffic  Control  (AT1-AT2)  Difficulty  contacting  ATC,  high  &  fast  clearances 


Baseline  Example  -  Survey  Results 


1)  Engine/Nacelle  smoke/fire/fumes  involving  electrical  components/wiring 

2)  Smoke,  fire  or  fumes  originated  in  cargo/baggage  area 

3)  Cargo/baggage  smoke/fire/fumes  involving  electrical  components/wiring 

4)  Passenger  compartment  smoke/fire/fumes  involving  electrical  components/wiring 

5)  Flights/attempted  flights  with  wrong  type  of  fuel 

6)  Severe  turbulence  resulting  in  occupant  injury 

7)  Airplane  went  off  end  of  runway 

8)  Hit/collided  with  run  way/taxi  way  lights 

9)  Hit  animal  other  than  bird 

10)  Collided/nearly  collided  with  ground  vehicle  on  ramp/apron 

11)  Nearly  collided  [with  another  aircraft]  while  on  runway 

12)  Began  takeoff  without  ATC  clearance  at  airport  with  active  control  tower 

13)  Near  collision  [with  terrain  or  ground  obstruction  while  airborne] 

14)  Airplane  crossed  runway  threshold  during  landing  approach  with  gear  up 

15)  Airplane  landed  with  gear  up 


Baseline  Example 


Acquisition  Example  -  How  do  I  design  a  safer  aircraft? 


Accident 

Reports 


OSHA 

Studies 


Aviation  Incident  Reports 


Baseline  Example 


Analysis  Example  -  How  do  I  react  in  the  event  of  a  crisis  at  an  airport? 


Reports 


Previous  Studies 


Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Set  Standards 
of 

Performance 


Design 

M&S 

Application 


Adjust  Throughout  the  Process! 


Simulation  Application  Requirement 


United  States  General  Accounting  Office 

GAO  Report  to  Congressional  Requesters 


November  1997  HUMAN  FACTORS 


FAA's  Guidance  and  Oversight  of  Pilot  Crew  Resource 
Management  Training  Can  Be  Improved 

GAO/RCED- 


Of  169  accidents  examined,  30  percent  were  caused  in  part  by  the  pilots’  performance. 
Pilots  incorrectly  used  the  principles  of  Crew  Resource  Management  (CRM)  -  an 
approach  to  improving  safety  through  coordination  of  the  cockpit  crew,  flight 
attendants,  dispatchers  and  air  traffic  controllers. 

Requirement  -  Train  pilots  and  crew  in  CRM 
Baseline  -  Results  from  GAO  report 
Standard  -  Reduce  percentage  of  accidents  due  to  CRM 


M&S  Application  -  Train  pilots  &  crew  in  CRM  through  simulation 


Determine  Requirements 
Example  -  Who  is  the  CRM  Audience? 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Pilot 


Flight 

Attendant 


Ground 

Crew 


TSA 


Ground  Air  Traffic 

Controller  Controller 


Local  Medical  On  Site 
Responders  Responders 


Ticket 

Agents 


Local  SWAT 
Teams 


Local  Police 
Aviation 


Medevac 

Teams 


Aircraft  crew  members  need  to  recognize,  evaluate  and  respond  to  emergency  situations 
on  an  aircraft  with  an  appropriate  crew  resource  management  course  of  action. 


Determine  Requirements 
Example  -  Who  is  the  CRM  Audience? 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Pilot 


Flight 

Attendant 


Ground  Air  Traffic 

Controller  Controller 


Aircraft  crew  members  need  to  recognize,  evaluate  and  respond  to  emergency  situations 
on  an  aircraft  with  an  appropriate  crew  resource  management  course  of  action. 


Design  M&S  Application 
Training  Example 


Two  Components  of  the  Simulation 


True  Physical  Environment 


Realistic  Scenario 


Simulation  Application  Options 


Establish  Alternatives 


Acquisition 


Training  Education 


Planning  Experimentation  T&E 


Live 

Simulation 


Constructive 

Simulation 


Virtual 

Simulation 


Simulation  Application  Options 


Establish  Alternatives 


Individual 


Individual  to  Small  Multiple 

Individual  Organization  Organizations 


Live  Simulation 


Constructive 

Simulation 


Virtual 

Simulation 


Design  M&S  Application 


Crisis  Management  Issues  -  Example 


Within  the  United  States  civilian  and  military  aviation  system  there 
are  numerous  crisis  situations  that  occur  each  year  that  receive  both 

local  and  national  attention. 


Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Set  Standards 
of 

Performance 


Design 

M&S 

Application 


Adjust  Throughout  the  Process! 


Design  Metrics 


Quantitative  -  Relating  to  the  measurement  of  quantity 


Qualitative  -  A  personal  understanding  that  is  based  on  qualities 


Foundational  Similarities: 


Comparing  Qualitative  and 
Quantitative  Methods 


(1)  All  qualitative  data  can  be  measured  and  coded  using  quantitative  methods. 

(2)  Quantitative  metrics  can  be  generated  from  qualitative  inquiries. 

Example:  Comparison  on  Realism:  How  real  does  it  look  to  you  on  a  scale  of  1-5? 


VS. 

Foundational  Differences: 

(1)  Difference  between  qualitative  &  quantitative  research  stems  from  underlying  strategies. 

(2)  Quantitative  research  is  viewed  as  confirmatory  and  deductive  in  nature. 


(3)  Qualitative  research  is  considered  to  be  exploratory  and  inductive. 


Design  Metrics 


Remember  the  perspective  of  who  the  ROI  is  for. 


Internal  to  the 
Program 


Program  Manager 
Using 

Organizations 


User 


Tragedy  at  Fort  Hood 


External  to  the 
Organization 


Press 

Other 


Environmentalist 


Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Set  Standards 
of 

Performance 


Design 

M&S 

Application 


Collect 

Data 


Adjust  Throughout  the  Process! 


Collect  Data 
Training  Example 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Collect  Data 
Training  Example 


Prior  to  the 
Simulation 
Exercise 

Relook  of 

Answers 

Post 

Simulation 

Results 

1 

Our  CRM  plan  addresses  communications. 

2 

I  am  prepared  for  a  crisis  situation. 

3 

I  make  decisions  with  input  from  others. 

4 

My  actions  would  be  consistent  in  a  crisis. 

5 

I  would  delegate  during  a  crisis. 

6 

I  would  keep  focused  through  resolution. 

7 

I  know  what  questions  to  ask  in  a  crisis. 

8 

I  know  how  to  execute  actions  in  a  crisis. 

9 

I  understand  the  implications  of  the  CRM  system. 

10 

Our  procedures  are  well  thought  out. 

11 

Contingencies  are  planned  for. 

12 

Our  emergency  response  system  is  effective. 

13 

I  know  how  to  take  charge  in  a  crisis. 

14 

I  am  prepared  to  handle  a  crisis. 

Collect  Data 
Training  Example 


Overall  Resource  Utilization 


1.  COMPONENT 
AIR  FORCE 


FY  2005  MILITARY  CONSTRUCTION  PROJECT  DATA 
(computer  generated) 


3.  INSTALLATION  AND  LOCATION 
AVIANO  AIR  BASE,  ITALY 


4.  PROJECT  TITLE 
PLIGHT  SIMULATOR 


5.  PROGRAM  ELEMENT 

6  CATEGORY  CODE 

7.  PROJECT  NUMBER 

8.  PROJECT  COST  ($000) 

27596 

171  212 

ASHE986013 

2.934 

COST  ESTIMATES 


TTM 


I' RIMARY  FACILITY 
FLIGHT  SIMULATOR 
ANTI -TERAORISM/FORCE  PROTECTION 
SUPPORTING  FACILITIES 
UTILITIES 
PAVEMENTS 
SITE  IMPROVEMENTS 
COMMUNICATION  SUPPORT 
PASSIVE  FORCE  PROTECTION 
SUBTOTAL 

CONTINGENCY  (  5.0  %) 

TOTAL  (0  STRUT  COST 

SUPERVISION,  INSPECTION  AND  OVERHEAD  (  6.5  %) 

TOTAL  REQUEST 

TOTAL  REQUEST  (ROUNDED) 

EQUIPMENT  FROM  OTHER  APPROPRIATIONS  (NON-ADD) 


560 

560 


1,926 
1,884) 
(  42  ) 

570 

(  160) 
(  140  ) 

<  204  ) 

(  250) 
(  10) 
2.496 
125 


2.621 

170 


2.792 

2.934 

(  36,045.0  ) 


10.  Description  of  Proposed  Construction:  Construct  facility  with  oversized  door  and 
removable  panels  for  equipment  chanqe  outs.  Support  space  for  attain,  training  support 
aachinery  and  maintenance  functions.  Simulator  room  will  have  raised  computer 
flooring,  soundproofing,  demolition,  and  all  utilities  required.  Provide  15  cm  (5.9a) 
concrete  pavements.  Must  comply  with  regional  tone  protection  standards. 


11.  REQUIREMENT:  1,349  SM  ADEQUATE:  0  SM  SUBSTANDARD:  157SM 

TROJECT:  Construct  a  new  flight  simulator  facility.  (Current  Mission) 

iEQUIREMENT;  Provide  Flight  Simulator  facility  to  support  aircrew  training  activities. 
The  project  mrti  include  adequate  space  for  administration  and  records,  classrooms,  and 
sther  supporting  spaces.  AT/FP  costs  on  this  project  are  higher  than  standard  Do£> 
juidance  due  to  stricter  EUCOM  force  protection  standards  requiring  screening  from 
lirect  fire  weapons. 

SITUATION:  The  current  flight  simulator  is  housed  in  one  building  with  the 


classrooms  and  administrative  spaces  in  separate  portable  buildings.  The  lack  of  an 
idequete  base  facility  for  this  requirement  represents  a  serious  deficiency  toward  pilot 

ceadiness. 

IMPACT  IF  NOT  PROVIDED:  Without  this  project,  vital  training  requirements  will  continue 
rp  be  performed  in  substandard  conditions  thus  seriously  compromising  pilot  readiness 
u»d  mission  in  the  Southern  European  Regions 

ADDITIONAL:  This  facility  is  eligible  for  NATO  funding.  The  NATO  funded  portion  ($1M) 

irovides  for  a  two-ship  facility.  This  US  cost  share  provides  the  other  two  ships,  560 
SH,  for  a  complete  four-ship  facility.  This  project  complies  with  space  criteria 
«utlined  in  AFH  32-1084,  "Facility  Requirements."  Project  requires  US/Italian  Mixed 


Labor  Hours 


Time  to  Input  Scenario  into  the  Simulation 

Item 

Hours 

Taskl 

20 

Task  2 

10 

Task  3 

12 

Task  4 

Time  to  Create  Scenario 

4 

Taskl 

40 

Task  2 

20 

Task  3 

38 

Task  4 

10 

One  Time  Exterior  Digitization  of  an  Airport 

Taskl 

142 

Task  2 

142 

Task  3 

121 

Total 

559  hours 

Process  for  Quantifying  M&S 


Determine 

Requirements 


Document  the  Process  and  Results! 


Set  Standards 
of 

Performance 


Design 

M&S 

Application 


Collect 

Data 


Adjust  Throughout  the  Process! 


Analyze  Data 


Analyze  Data 
Training  Example 


Training  Example  -  How  do  I  prepare  employees  for  an  aviation  incident  /accident? 


Analyze  Data 
Training  Example 


Prior  to  the 
Simulation 
Exercise 

Relook  of 
Answers 

Post 

Simulation 

Results 

1 

Our  CRM  plan  addresses  communications. 

5 

3.1 

3.6 

2 

I  am  prepared  for  a  crisis  situation. 

4.9 

4.1 

4.1 

3 

I  make  decisions  with  input  from  others. 

4.2 

3.2 

3.2 

4 

My  actions  would  be  consistent  in  a  crisis. 

4.9 

3.2 

3.2 

5 

I  would  delegate  during  a  crisis. 

4.9 

2.7 

4.3 

6 

I  would  keep  focused  through  resolution. 

4.1 

3.1 

3.2 

7 

I  know  what  questions  to  ask  in  a  crisis. 

4.4 

2.2 

2.3 

8 

I  know  how  to  execute  actions  in  a  crisis. 

4.9 

3.1 

3.1 

9 

I  understand  the  implications  of  the  CRM  system. 

5 

2.4 

4.3 

10 

Our  procedures  are  is  well  thought  out. 

4.9 

1.5 

3.9 

11 

Contingencies  are  planned  for. 

5 

1.5 

3.9 

12 

Our  emergency  response  system  is  effective. 

4.9 

2.7 

3.9 

13 

I  know  how  to  take  charge  in  a  crisis. 

4.6 

3.1 

3.1 

14 

I  am  prepared  to  handle  a  crisis. 

4.9 

1.7 

3.5 

Analyze  Data 
Training  Example 


Results  of  Pre/Post  Survey 


1  3  5  7  9  11  13  15  17  19  21  23  25 


Question  Number 


Series  1 


Series2 


^ —  Series3 


Series  1  -  Pre-simulation  answers 
Series  2  -  Post  simulation 
Series  3-6  Months  Post  simulation 

Line  between  points  are  to  show  a  trend  &  do  not  indicate  continuity 


Analyze  Data 
Training  Example 


Actions  to  be  Taken  by  the  Staff  Based  on  the  Simulation  Exercise: 

Information  was  derived  from  multiple  transcripts  that  were  created  from  the 
videotaping  of  the  simulation  exercise  sessions  and  after-action  reviews. 

•  Modified  crew  hours  and  legs  by  year 

•  Changed  crew  activities  during  aircraft  icing 

•  Increased  awareness  and  information  about  bird  strikes 

•  Changed  procedures  to  avoid  collision  with  ground  vehicle 

•  Increased  training  with  ground  proximity  warning  system 

•  Developed  scenarios  to  practice  water  landings 

•  Redefined  CRM  for  medical  issues  &  passenger  disturbance 

•  Increased  training  for  near  midair  collisions 


Analyze  Data 
Training  Example 


During  one  iteration  of  simulation  runs  and  after-action  review  session,  the  various 
crews  identified  38  actions  that  needed  to  be  addressed. 


Actions  ranged  from  a  simple  policy  change  to  a  complete 
revision  of  some  aspects  of  CRM  plans  and  procedures. 


Example  of  one  action  -  What  technical  information  is  available  to  assist  both  the  crew  and  ground 
support  advisors? 

•  Several  crew  members  felt  a  number  of  informational  items  were  distractors  and 
unimportant. 

•  Ground  controllers  explained  why  items  are  extremely  important— timing  & 
location  aspects . 


Analyze  Data 
Training  Example 


Overall  Resource  Utilization 


1.  COMPONENT 
AIR  FORCE 


FY  2005  MILITARY  CONSTRUCTION  PROJECT  DATA 
(computer  generated) 


3.  INSTALLATION  AND  LOCATION 
AVIANO  AIR  BASE,  ITALY 


4.  PROJECT  TITLE 
PLIGHT  SIMULATOR 


5.  PROGRAM  ELEMENT 

6.  CATEGORY  CODE 

7.  PROJECT  NUMBER 

8.  PROJECT  COST  ($000) 

27596 

171212 

ASHE986013 

2.934 

COST  ESTIMATES 


*’  RIMARY  FACILITY 
FLIGHT  SIMULATOR 
ANTI -TERROR ISM/ FORCE  PROTECTION 
SUPPORTING  FACILITIES 
UTILITIES 
PAVEMENTS 
SITE  IMPROVEMENTS 
COMMUNICATION  SUPPORT 
PASSIVE  FORCE  PROTECTION 
SUBTOTAL 

CONTINGENCY  (  $.0  %) 

TOTAL  CONTRACT  COST 

SUPERVISION,  INSPECTION  AND  OVERHEAD  (  6.5  %) 

TOTAL  REQUEST 

TOTAL  REQUEST  (ROUNDED) 

EQUIPMENT  FROM  OTHER  APPROPRIATIONS  (NON-ADD) 


560 

560 


1,926 
1,884) 
(  42) 
570 

(  160) 
(  240  ) 
(  204  ) 
(  ISO) 
(  10) 
1.496 
125 


2.621 

170 


2.792 

2.S34 

(  36,045.0  ) 


10.  Description  of  Proposed  Construction:  Construct  facility  with  oversized  door  and 
enoveble  panels  for  equipment  change  outs.  Support  space  for  aefeoin.  training  support 
echinery  and  maintenance  functions.  Simulator  room  will  have  raised  computer 
looring,  soundproofing,  demolition,  and  all  utilities  required.  Provide  15  cm  (5.9*) 
concrete  pavements.  Must  comply  with  regional  twee protection  standards. 


11.  REQUIREMENT:  1,349  SM  ADEQUATE:  0  SM  SUBSTANDARD:  157SM 

PROJECT:  Construct  a  new  flight  simulator  facility.  (Current  Mission) 

REQUIREMENT ;  Provide  Flight  Simulator  facility  to  support  aircrew  training  activities. 
The  project  mmi  include  adequate  space  for  administration  and  records,  classrooms,  and 
ether  supporting  spaces.  AT/FP  costs  on  this  project  are  higher  than  standard  Do£> 
juidance  due  to  stricter  EUCOM  force  protection  standards  requiring  screening  from 
lirect  fire  weapons. 

SITUATION:  The  current  flight  simulator  is  housed  in  one  building  with  the 


classrooms  and  administrative  spaces  in  separate  portable  buildings.  The  lack  of  an 
idequate  base  facility  for  this  requirement  represents  a  serious  deficiency  toward  pilot 

;«adinesa. 

IMPACT  IF  NOT  PROVIDED:  Without  this  project,  vital  training  requirements  will  continue 
tp  be  performed  in  substandard  conditions  thus  seriously  compromising  pilot  readiness 
tnd  mission  in  the  Southern  European  Regions 

1DPIT ZONAL:  This  facility  is  eligible  for  NATO  funding.  The  NATO  funded  portion  ($1M) 
trovidea  for  a  two-ship  facility.  This  US  cost  share  provides  the  other  two  ships,  560 
IM,  for  a  complete  four-ship  facility.  This  project  complies  with  space  criteria 
outlined  in  AFH  32-1084,  "Facility  Requirements . "  Project  requires  US/Italian  Mixed 


Time  to  Create  Each  Scenario 


Item 

Costs 

Task  1  (40  hrs) 

$2800 

Task  2  (20  hrs) 

$3000 

Task  3  (38  hrs) 

$1900 

Task  4  (10  hrs) 

$400 

Total 

$8100 

One  Time  Exterior  Digitization  of  an  Airport 

Item 

Costs 

Task  1  (142  hrs) 

$18886 

Task  2  (142  hrs) 

$11502 

Task  3  (121  hrs) 

$8288 

Total 

$38316 

Time  to  Input  Scenario  into  the  Simulation 

Item 

Costs 

Task  1  (20  hrs) 

$560 

Task  2  (10  hrs) 

$280 

Task  3  (12  hrs) 

$360 

Task  4  (4  hrs) 

$160 

Total 

$1360 

Total  Cost:  $36,092,776 


Process  for  Quantifying  M&S 
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Document  the  Process  and  Results! 
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of 
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Design 
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Application 
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Adjust  Throughout  the  Process! 


Determine  ROI 

t 


Analyze  Data 


ROI  Example  -  Training 


Multiple  Configurations 


M 

* 

Individual  Trainer  Team  Role 
Configuration  Trainer 


Seminar 

Configuration 


Conference 

Configuration 


Rehearsal 

Configuration 


Situation  Tool 


Analysis 

Configuration 


ROI  Example  -  Training 


•  Efficiency  for  the  232,271  individuals  in  USA  required  to  receive  CRM  training 

•  43,681  airline  transport  pilots 

•  78,670  flight  attendants 

•  29,430  air  traffic  controllers 

•  Prior  to  1998  CRM  was  optional  for  airlines 

•  FAA  provided  two  methods  for  CRM  training — classroom  or  simulator 

•  GAO  -  CRM  just  taught  in  a  classroom  without  experiential  experience  is  less  effective 

•  Simulation  can  run  any  time 

•  Now  FAA  mandated  training 


450  Commercial  Airports 

Numerous  Types  of  Commercial  Aircraft 

Each  individual  can  be  trained  in  their  own  environment 


ROI  Example  -  Training 


During  the  one  simulation  and  after-action  review  session,  the  staff 
identified  38  actions  that  needed  to  be  addressed.  All  38  were  addressed 

which  led  to  an  additional  24  actions. 

Actions  ranged  from  a  simple  policy  change  to  a  complete  revision  of  the  plans. 

•  Changed  the  notification  to  the  press  procedures 

•  Changed  plan  so  that  an  emergency  on  an  aircraft  mobilizes  the  entire  community 

•  Created  a  responsibility  list  for  all  administrators  to  follow  while  incident  is  occurring 

•  Changed  procedures  for  dealing  with  passengers 

•  Further  delineated  responsibilities  between  pilots,  flight  attendants  &  air  traffic 
controllers 

Over  the  last  ten  years  since  the  FAA  mandated  simulation-based  CRM  training, 
there  have  been  382  reported  incidents  with  only  12  incidents  caused  by  improper 

use  of  CRM. 


ROI  Example  -  Training 


Number  of  incidents  and  fatalities  during  the  last  ten  years 


Incidents 

98 

99 

01 

02 

03 

04 

05 

07 

08 

Total 

Number 

33 

26 

25 

34 

34 

45 

33 

42 

37 

37 

28 

8 

382 

CRM  Related  Cause 

0 

2 

0 

3 

1 

1 

1 

0 

2 

1 

0 

1? 

12 

Fatalities 

1 

12 

92 

531 

0 

22 

14 

22 

50 

1 

0 

49 

794 

%  CRM  Related 

3% 

-Everything  that  could  go  wrong  on  that  flight  to  Buffalo  on  that 
wintery  night  did.  We  had  a  pilot  who  had  not  been  trained  on 
how  to  handle  emergency  procedures.  He  had  never  received 
simulator  training  for  stall  warnings,  and  reacted  exactly 
opposite  the  way  he  should  have,  pulling  back  on  the  stick  rather 
than  pushing  it  forward  to  increase  airspeed.  It's  a  fatal  mistake.” 

Former  FAA  Chief  Michael  Goldfarb 

NSTB  Hearing  May  12,  2009  -Colgan  Air,  Inc.  Flight  3407 


2009  New  York  Crash 


ROI  Example  -  Training 


American  Airlines 
Example 


ROI  Example  -  Training 


American  Airlines 
Example 


Simulators  Usage  18-20  hours  per  day  -  917 
Train  9,000  American  Airlines’  pilot  workforce 
Re-certification  at  least  every  nine  months 
12  hours  of  simulation  time  per  pilot 
Over  10  years,  13  times  per  pilot  -  156  hours 
Required  re-certification  -  936,000  hours 
Initial  certification,  changing  seat  or  aircraft  - 
Sell  simulation  time  at  $500  per  hour;  100,000 


,280  hours  per  year  (2  positions  per  hour) 


230,000  hours 
hours  -  $50  million 


ROI  Example  -  Training 


r 


American  Airlines 
Example 


v 


Ten  Year  Costs  $153  million 


Per  hour  for  a  737  -  $2,200 


Required  re-certification  -  268,000  flight  hours 


/ 


L  ,  4 


Time 


Money 


s 


a 


ROI 

Per  Year  Savings 

$179,600,000 


Initial  certification,  changing  seat  or  aircraft  -  115,000  flight  hours 


Total  -  583,000  flight  hours  X  $2,200  =  $1,282,600,000 


Overall  ROI  -  $1,282,600,000  -  $103,000,000  (less  $50M)  =  $1,179,600,000 


ROI  Example  -  Training 


Situation  Tool 


20  Minutes 


10  Minutes 

BP* ' 


Wind 

Direction  and 
Speed 


— 


Environment 


Process  for  Quantifying  M&S 
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If  Needed,  Initiate 
Corrective  Action 

1 

Determine  ROI 

Analyze  Data 

Contents  of  an  ROI  Evaluation  Plan 


Document  the  Process  and  Results! 
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